Estimation of Position Coordinates of Light Sources in Images of the Surrounding Environment

ABSTRACT

The invention relates to a device and a method for estimating position coordinates and the type of light sources in images of the surrounding environment, in particular in images of the surrounding environment of a surround-view camera in vehicles. The type of light source (120, 125, 128) can be a spot light source or a direction light source. The method comprises the steps:a) capturing a surrounding environment (100) in the form of an image (300) of the surrounding environment by means of a camera (200);b) determining areas of increased brightness (310) of the image (300) of the surrounding environment, specifically of an amount of pixels, the brightness of which exceeds a predefined threshold;c) determining position coordinates (335) of the light source (120, 125, 128) from the areas of increased brightness (310) and according to a first calculation method;d) determining the type of light source (120, 125, 128) from the shadow (350) of an object (150), in particular a vehicle, on the image (300) of the surrounding environment.

The invention relates to a device and a method for estimating positioncoordinates and the type of light sources in images of the surroundingenvironment, in particular in images of the surrounding environment of asurround-view camera in vehicles. Furthermore, the invention relates toa program element and a computer-readable medium.

Methods for estimating position coordinates of light sources in imagesof the surrounding environment are, for example, used in order tocompile an image which is as realistic as possible of the surroundingenvironment in so-called augmented reality applications. More recently,these methods are used in particular in land vehicles, in order toconvey a rapid and intuitive image of his surrounding environment to thedriver, in which the elements of augmented reality are displayed. Thatis to say, a photo-realistic image of the surrounding environment isrequired, which is calculated in real time—including at high speeds ofthe vehicle. The estimation of position coordinates and the type oflight sources in images of the surrounding environment play a particularrole in this. Two- or three-dimensional models of the location of eachlight source are understood to be position coordinates. In a number ofembodiments, position coordinates are understood to be the direction ofeach light source. Spot light sources or direction light sources areunderstood to be the type of light source. Direction light sources(“directional illumination sources”) are typically very distant lightsources, e.g. the sun. Spot light sources (“point illumination sources”)are light sources which are arranged closer, e.g. street lamps.

Methods in the prior art require a very high amount of computationalpower for these calculations, Furthermore, a plurality of methods areonly able to consider a single light source which is deemed to be a mainlight source. Many methods additionally require reference objects inorder to estimate position coordinates of one or more light sources.

Against this backdrop, it is an object of the invention to at leastpartially overcome the disadvantages of the prior art.

This object is achieved by the subject-matter of the independent claims.Further developments of the invention are set out in the sub-claims andthe following description.

The invention comprises a method for estimating position coordinates andthe type of a light source in images of the surrounding environment,wherein the type of light source can be a spot light source or adirection light source. The method comprises the following steps:

a) Capturing a surrounding environment in the form of an image of thesurrounding environment by means of a camera. In a number ofembodiments, the image of the surrounding environment is, in particularin the case of moving land vehicles, acquired at regular temporalintervals. The vehicle can move between the individual acquisitions. Ina number of embodiments, this is considered in that the calculatedaugmented reality images are interpolated in a suitable manner, inparticular considering the speed of the vehicle. The described method isapplied to each individual image of the surrounding environment. Theimage of the surrounding environment is temporarily stored, for example,in a primary memory of a control unit or of a computer. In a number ofembodiments, this computer is located in the vehicle or is connected tothe vehicle.

b) Determining areas of increased brightness of the image of thesurrounding environment, specifically of an amount of pixels, thebrightness of which exceeds a predefined threshold. The predefinedthreshold can be fixed or variable. In an embodiment, the threshold ofthe total brightness or the maximum brightness of the image of thesurrounding environment is determined. This can in particular be thecase with images of the surrounding environment, which have beenacquired after sundown.

The captured image of the surrounding environment is subsequentlyanalyzed as to whether the amount of pixels is contiguous ordiscontiguous. If the amount of pixels is contiguous, this is thendeemed to be a (contiguous) area of increased brightness. If the amountof pixels is discontiguous, it is then assumed that the captured imageof the surrounding environment has multiple separate light sources. Ifthis is the case, the following steps are then applied to eachindividual light source. In a number of embodiments, a minimum distancebetween discontiguous pixels is predefined, so that discontiguous pixelsonly lead to the assumption of multiple, separate light sources as ofthis minimum distance.

c) Determining position coordinates of the light source from the areasof increased brightness, according to a first calculation method. Theareas of increased brightness have an edge which is determined by thepredefined threshold of the brightness. In a number of embodiments, theedge between a pixel which exceeds the threshold and a neighboring pixelwhich falls short of the threshold can be determined. In the case of anumber of embodiments, the edge can be smoothed.

Following the calculation of the edge of the areas of increasedbrightness, according to a first calculation method, the middle of theareas of increased brightness is determined. The first calculationmethod can, for example, determine the geometric center (centroid) ofeach area of increased brightness or can use a secant method or anothermethod in order to calculate the midpoint of each area. This midpoint orcenter is viewed as position coordinates of the light source.

d) Determining the type of light source from the shadow of an object, inparticular a vehicle, on the image of the surrounding environment. Aftercalculating the position coordinates of each light source, the shadowsof the image of the surrounding environment are analyzed. The camera canbe arranged on the object which is casting shadows. This object can inparticular be a vehicle. A shadow is assigned to each light source.Furthermore, the type of light source is determined from the form of theshadow, for example from the parallelism of the edges.

This type of calculation is particularly advantageous, because theresults thus obtained, specifically the knowledge of both the positioncoordinates of each light source and their type, lead to a particularlyrealistic modelling of augmented reality images. It is, in addition,advantageous that not only can a single main light source be consideredwith this method, but also multiple light sources. In particular, anomnidirectional image of the surrounding environment is no longernecessarily required as a basis for this method. Furthermore, theseresults can be obtained with a lower computational cost—i.e. a shortercomputation time or with a slower processor. Elaborate modellings of thesurrounding environment and/or the use of databases with known objectsis/are additionally dispensed with.

In an embodiment, the invention has an additional step: e) Applying asmoothing filter, in particular a Kalman filter, to the positioncoordinates of the light source.

This is in particular advantageous in order to at least partiallycompensate for erroneous estimations from the preceding method steps.This step can additionally be advantageously utilized if a (e.g.temporal) sequence of images of the surrounding environment exists, e.g.if the camera is arranged on or in a vehicle which is moving. A furtheradvantage of the method is that it can also be used to calculatenon-static scenes.

In an embodiment, the camera consists of a plurality of cameras and isin particular a surround-view camera. For example, four cameras can bearranged on or in a vehicle in an embodiment. The arrangement canprovide one camera each for each side (on the left, on the right, at thefront, at the rear). The cameras do not have to be arranged at an angleof precisely 90°. The advantage of this embodiment is that the positioncoordinates of the light source can be calculated as three-dimensionalcoordinates. In addition, the cameras can be used in their overlappingregion in order to correct the image of the surrounding environment ofthe other camera in each case.

In an embodiment, one of the steps b), c), d) or e) is carried outmultiple times. In an embodiment, these steps are applied to amultiplicity of light sources.

In an embodiment, the areas of increased brightness of the image of thesurrounding environment are white-saturated. The value “white-saturated”is the highest brightness value of the color scale used, that is to sayfor example the hexadecimal value “FF” in the case of a pure grayscaleor “FFFFFF” in the case of a RGB scale. These values are avoided in theprior art in some algorithms, because these values can lead todifficulties in some algorithms and/or can cause higher computationtimes. However, in the case of the present invention, the use of thevalue “white-saturated” leads to a simplification—and, thus, to afurther acceleration—of the calculations.

In an embodiment, step c) has the following sub-steps:

c1) selecting three pixels at the edge of the areas of increasedbrightness;

c2) connecting two pixels each, from the three pixels, by a first and asecond straight line;

c3) forming a first and a second normal through the middle of the firstand the second straight line;

c4) determining the position coordinates of the light source from anintersection of the first and the second normal.

In an embodiment, step d) has the following sub-steps:

d1) assigning the shadow to the light source from the edges of theshadow of the object and the position coordinates of the light source.

d2) determining, from the parallelism of the edges of the shadow, thetype of light source, specifically a direction light source fromparallel edges and a spot light source from non-parallel edges.

The invention also comprises a control unit which is connected to thecamera and which is designed to carry out the indicated method. If thecamera or the cameras is/are located on a vehicle, the control unit canlikewise be located in the vehicle. However, a part of the computationalpower can also be outsourced, e.g. to an external server.

The invention also comprises a program element which, if it is run onthe indicated control unit, instructs the control unit to carry out theindicated method.

The invention moreover comprises a computer-readable medium, on whichthe indicated program element is stored.

The invention is described below on the basis of drawings of specificembodiments which, it should be understood, are primarily intended toclarify, but not restrict, the invention, wherein:

FIG. 1a schematically shows a possible setting, in which the indicatedmethod is applied;

FIG. 1b schematically shows an example of calculating positioncoordinates of a light source from an image of the surroundingenvironment;

FIG. 1c schematically shows an example of calculating the type of alight source;

FIG. 2 shows an example of an image of the surrounding environment whichhas been acquired with a surround-view camera;

FIG. 3 shows an example of a method for calculating position coordinatesand the type of a light source.

FIG. 1a schematically shows an overview of a possible setting, in whichthe indicated method is applied. The surrounding environment 100 hasthree light sources 120, 125, 128. The light source 120 is sketched asthe sun, i.e. as a direction light source, the light sources 125 and 128are sketched as spot light sources. Furthermore, the surroundingenvironment 100 has a vehicle 150, on which four cameras 200 arearranged. A control unit 400 is additionally arranged in the vehicle150, which control unit is connected to the cameras via one connection420 each. The vehicle 150 casts a shadow 350 which is caused by thefirst light source 120. The shadows of the second light sources 125 and128 are not represented here.

FIG. 1b shows an example of calculating position coordinates of a lightsource from an image 300 of the surrounding environment, which has beenacquired by the four cameras 200. The cameras 200 are not usuallyvisible in an image 300 of the surrounding environment; these have onlybeen inserted here for illustration purposes. One way of calculatingposition coordinates is shown using the first light source 120. Here,the areas of increased brightness 310 of the first light source 120have, for example, the value “white-saturated”. Three pixels 331, 332,333 are selected from the edge 330 of the areas of increased brightness310 and two of them are each connected with straight lines 341, 342. Theperpendicular is dropped onto the middle of these straight lines 341,342 or the normals 346, 347 are constructed. The intersection 335 of thenormals 346 and 347 is the calculated middle 335 of the areas ofincreased brightness 310. In connection with this, the fact that twocameras 200 have captured the first light source 120 can advantageouslybe utilized. This means that it is possible to determinethree-dimensional position coordinates of the first light source 120.

This method can also be applied to the second light sources 125 and 128.The position coordinates of the light sources 120, 125, 128 aretherefore calculated from the image 300 of the surrounding environmentin FIG. 1 b.

FIG. 1c shows an example of calculating the type of a light source fromthe image 300 of the surrounding environment. The shadow 350 is firstassigned to the first light source 120, the midpoint 335 of which hasalready been calculated. The shadow 350 has the edges 351, 352. It isclearly visible that the edges 351, 352 are parallel. The result of thecalculation is that the first light source 120 is a direction lightsource.

The shadow which is cast by the second light source 125—and, similarly,also by the second light source 128, obviously has different edges 355,356. Since these edges 355, 356 are not parallel and the edges 355, 356can be assigned to the second light source 125, this second light source125 can be identified as a spot light source. Similarly, the secondlight source 128 is also identified as a spot light source (shadow notrepresented).

FIG. 2 shows an example of an image 300 of the surrounding environment,which has been acquired with a surround-view camera. A first lightsource 120 and a second light source 125 are clearly visible. The firstlight source 120 has a brightness value “white-saturated”. It is alsovisible that the areas of increased brightness, which are formed by thefirst light source 120, do not necessarily have to be a circle, in orderto be able to carry out the first calculation method. Furthermore, avehicle 150 is visible in the image 300 of the surrounding environment.The vehicle 150 casts a shadow 350 which is caused by the first lightsource 120. The shadow 350 has the edges 351, 352. It is clearly visiblethat the edges 351, 352 are parallel. In addition to the positioncoordinates of the light source, the calculation therefore reveals thatthe first light source 120 is a direction light source.

The shadow which is cast by the second light source 125 has the edges355, 356. Since these edges 355, 356 are not parallel and these edges355, 356 can be assigned to the second light source 125, said secondlight source 125 can be identified as a spot light source.

FIG. 3 shows an example of a method for calculating position coordinatesand the type of a light source.

A surrounding environment 100 in the form of an image 300 of thesurrounding environment is captured in a first step 501. This iseffected by means of a camera 200. Said camera 200 can also be executedas a surround-view camera and can comprise multiple physical cameras 200which are subsequently combined to produce a total image 300 of thesurrounding environment.

In a second step 502, the areas of increased brightness 310 of the image300 of the surrounding environment are determined, specifically anamount of pixels, the brightness of which exceeds a predefinedthreshold. In an embodiment, the areas of increased brightness 310 ofthe image of the surrounding environment can be white-saturated. Ifmultiple light sources 120, 125, 128 are imaged in the image 300 of thesurrounding environment, then the areas of increased brightness 310 aredetermined for each individual light source 120, 125, 128.

In a third step 503, the position coordinates 335 of each individuallight source 120, 125, 128 are determined from the areas of increasedbrightness 310, according to a first calculation method. The firstcalculation method can, for example, determine the geometric center ofeach area of increased brightness or can use a secant method or anothermethod for calculating the midpoint of each area. This midpoint orcenter is viewed as the position coordinates of the light source.

In a fourth step 504, the type of light source 120, 125, 128 isdetermined from the shadow 350 of a vehicle 150. That is to say, afterthe position coordinates of each light source are calculated, theshadows of the image of the surrounding environment are analyzed. Thecamera can be arranged on the object which is casting shadows. Thisobject can in particular be a vehicle. A shadow is assigned to eachlight source 120, 125, 128. Furthermore, the type of light source isdetermined from the form of the shadow, for example from the parallelismof the edges.

In addition, it is pointed out that “comprising” and “having” do notexclude any other elements or steps and the indefinite article “a” doesnot exclude a plurality. It is additionally pointed out that features orsteps, which have been described with reference to one of the aboveexemplary embodiments, can also be used in combination with otherfeatures or steps of other exemplary embodiments described above.Reference numerals in the claims are not to be deemed to berestrictions.

LIST OF REFERENCE NUMERALS

-   100 Surrounding environment-   120 First light source-   125, 128 Second light sources-   150 Vehicle-   200 Camera-   300 Image of the surrounding environment-   320, 322, 324 Areas of increased brightness-   330 Edge of the areas of increased brightness-   331, 332, 333 Pixel from the edge-   335 Calculated middle of the areas of increased brightness-   341, 342 Straight lines-   346, 347 Normals-   350 Shadow of the first light source-   351, 352 Edge of the shadow of the first light source-   355, 356 Edge of the shadow of the second light source-   400 Control unit-   420 Camera—control unit connection-   501 to 504 Steps of the method

1. A method for estimating position coordinates and the type of a lightsource (120, 125, 128) in an image (300) of the surrounding environment,wherein the type of light source (120, 125, 128) can be a spot lightsource or a direction light source, having the steps: a) capturing asurrounding environment (100) in the form of an image (300) of thesurrounding environment by means of a camera (200); b) determining areasof increased brightness (310) of the image of the surroundingenvironment (300), specifically of an amount of pixels, the brightnessof which exceeds a predefined threshold; c) determining positioncoordinates (335) of the light source (120, 125, 128) from the areas ofincreased brightness (310), according to a first calculation method; d)determining the type of light source (120, 125, 128) from the shadow(350) of an object (150), in particular a vehicle, on the image of thesurrounding environment (300).
 2. The method according to claim 1,having the additional step: e) applying a smoothing filter, inparticular a Kalman filter, to the position coordinates of the lightsource (120, 125, 128).
 3. The method according to claim 1, wherein thecamera (200) consists of a plurality of cameras and is in particular asurround-view camera.
 4. The method according to claim 1, wherein one ofthe steps b), c), or d) is carried out multiple times and is applied toa multiplicity of light sources (120, 125, 128).
 5. The method accordingto claim 1, wherein the areas of increased brightness (310) of the imageof the surrounding environment are white-saturated.
 6. The methodaccording to claim 1, wherein step c) has the following sub-steps: c1)selecting three pixels (331, 332, 333) at the edge of the areas ofincreased brightness (310); c2) connecting two pixels (331, 332, 333)each, from the three pixels (331, 332, 333), by a first (341) and asecond (342) straight line; c3) forming a first (346) and a second (347)normal through the middle of the first (341) and the second (342)straight line; c4) determining, from an intersection (335) of the first(346) and the second (347) normal, the position coordinates (335) of thelight source (120, 125, 128).
 7. The method according to claim 1,wherein step d) has the following sub-steps: d1) assigning the shadow(350) to the light source (120, 125, 128) from the edges (351, 352, 355,356) of the shadow (350) of the object (150) and the positioncoordinates (335) of the light source (120, 125, 128); d2) determining,from the parallelism of the edges (351, 352, 355, 356) of the shadow(350), the type of light source (120, 125, 128), specifically adirection light source from parallel edges (355, 356) and a spot lightsource from non-parallel edges (355, 356).
 8. A control unit (400) whichis connected to a camera (200) and which is configured to carry out themethod according to claim
 1. 9. A program element which, if it is run ona control unit (400), instructs the control unit (400) to carry out themethod according to claim
 1. 10. A computer-readable medium, on which aprogram element according to claim 9 is stored.